Serverless App using AWS API, DynamoDB, Lambda, S3 and Visual Studio .Net

This is a sample project using Visual Studio 2017 (.Net 4.5 + Core 2) and the following AWS services:

API Gateway
DynamoDB
Lambda
S3
CloudFormation

This project will have an Angular web front end hosted on S3, which calls APIs in the API Gateway. Those APIs are defined as Lambda functions and interact with DynamoDB as its data source.

Github Repository

The full project repository can be found here on github:

https://github.com/johnlee/Widgets.AWS.Serverless

Setting up AWS Account

Set up a developer group and account to be used programmatically by the app. This is done through IAM. The following built-in policies were added to the developer group.

AWSLambdaFullAccess
AmazonS3FullAccess
AmazonAPIGatewayInvokeFullAccess
AmazonDynamoDBFullAccess
AWSLambdaDynamoDBExecutionRole
AWSCodeDeployRoleForLambda
AWSCloudFormationReadOnlyAccess

On top of the built-in policies, we need a couple of inline policies. Create and attach the following policies to the group.

cloudformation

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1516844776000",
            "Effect": "Allow",
            "Action": [
                "cloudformation:*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

iam

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1516916709000",
            "Effect": "Allow",
            "Action": [
                "iam:AttachRolePolicy",
                "iam:CreateRole",
                "iam:DetachRolePolicy"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

manage.apigateway

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1516917382000",
            "Effect": "Allow",
            "Action": [
                "apigateway:*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

A service user account was created and assigned to the group created above. This user account has its own access keys, which are saved/downloaded as a csv file. For IAM best practices refer to this article:

https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html

Setting up the Visual Studio

The following Visual Studio plugins needed:

Nodejs for Visual Studio
https://github.com/Microsoft/nodejstools/releases/tag/v1.3.1
AWS Toolkit for Visual Studio
https://aws.amazon.com/visualstudio/

We need to setup Visual Studio to use the AWS SDK that was downloaded above. Install the AWS Visual Studio SDK. After installation, a new explorer view is available in Visual Studio called “AWS Explorer”. This can be found under the “View” menu.

Next we need to create a profile in the AWS Explorer. This profile will be used by the AWS projects. Click on the new profile button to configure the profile. Enter the user account that was created in IAM in the section above.

The important thing to note is that the profile uses an account that is not the root. As noted in the section above, it is best to create a group with listed policies above and assign the user to that group.

Project Implementation

I’ve created a new .NET Console Application in Visual Studio called Widgets. Refer to the GitHub link for actual project. In the Widgets application I’ve created a console project which will be used to create and setup the DynamoDB database. The following NuGet packages were installed.

AWSSDK.Core (v3.3.21)
AWSDynamoDBv2 (v3.3.5)

Once the packages are installed, I added the AWS credentials to the appsettings so that the project will use the profile we configured above. This information can be configured in the app.config like so:

  <appSettings> 
   <add key="AWSProfileName" value="myawsprofile"/> 
   <add key="AWSRegion" value="us-east-2" />
  </appSettings>

In the solution, there are 4 projects in total. The first two were used to initialize the DynamoDB and used classical .Net (version 4.5). The last two projects are related to the actual running application and use .Net Core (version 2). The Widgets.Lambda and Widgets.Web are the cores of the application.

Widgets (console app for setting up DynamoDB)
Widgets.Data (used by the console app)
Widgets.Lambda (web api using .Net Core)
Widgets.Lambda.Tests (tests for lambda)
Widgets.Web (web front end)

Setup DynamoDB

The main console project has a class for setting up the dynamoDB called SetupDynamoDB.cs. There are three methods for creating, deleting and updating tables. Sample code for creating and deleting shown below. Refer to main github project for full code source.

        
public static void CreateTables()
{
    // Create Widget table if not exist
    List currentTables = _client.ListTables().TableNames;
    if (!currentTables.Contains(_tableNameWidget))
    {
        // Create table request
        var request = new CreateTableRequest
        {
            TableName = _tableNameWidget,
            KeySchema = new List()
            {
                new KeySchemaElement
                {
                    AttributeName = "Id",
                    KeyType = KeyType.HASH
                }
            },
            AttributeDefinitions = new List()
            {
                new AttributeDefinition
                {
                    AttributeName = "Id",
                    AttributeType = ScalarAttributeType.N
                }
            },
            ProvisionedThroughput = new ProvisionedThroughput
            {
                ReadCapacityUnits = 10,
                WriteCapacityUnits = 5
            }
        };

        // Create the table
        var response = _client.CreateTable(request);
    }
}

public static void DeleteTables()
{
    List currentTables = _client.ListTables().TableNames;
    if (currentTables.Contains(_tableNameWidget))
    {
        _client.DeleteTable(_tableNameWidget);
    }
}

More information about working with DynamoDB tables can be found here:

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/LowLevelDotNetWorkingWithTables.html

There is some sample data pre-populated into the tables. These can be found in the console project in the SeedDynamoDB.cs class. This class uses the Data project to perform the CRUD functions. Some sample code below on inserting, updating and deleting data. Refer to main github project for full example. Note that when working with DynamoDB the data format is JSON.

public static void AddSampleData()
{
    Repository repository = new Repository(_client);

    // Types
    List types = new List();
    foreach (var typeId in _typeIds)
    {
        types.Add(new Data.Type
        {
            Id = typeId,
            Name = $"WDGT_TYPE_{typeId}",
            Description = $"Widget Type {typeId}"
        });
    }
    repository.Types.AddBatch(types);
}

public static void UpdateSampleData()
{
    Repository repository = new Repository(_client);
    var types = repository.Types.GetAll(_typeIds);
    foreach (var type in types)
    {
        type.Name = "Type " + type.Id;
        repository.Types.Update(type.Id, type);
    }
}

Note there are two abstract layers provided by AWS to interact with the data. For persisted object bind we can use DynamoDBContext or for a lower layer document level control we have the AWS DocumentModel. For more details on each of the approaches see the links below:

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DotNetSDKHighLevel.html

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DotNetSDKMidLevel.html

Setup Lambda

Every Lambda function requires an IAM role associated with it. The role defines what AWS services the function is allowed to interact with. This includes interactions with DynamoDB and CloudWatch (logging). For an example of setting up a Lambda to DyanmoDB role, refer to section 2 of the following article:

https://github.com/awslabs/aws-serverless-workshops/tree/master/WebApplication/3_ServerlessBackend

With Lambda and AWS API Gateway we are able to run a typical ASP.NET Web API application by replacing the web server as so:

The AWS Toolkit for Visual Studio provides a project template for Web API on Lambda. The whole Web API project can be deployed into S3. Details of this can be found here:

https://docs.aws.amazon.com/lambda/latest/dg/lambda-dotnet-create-deployment-package-toolkit.html

In this solution the Widgets.Lambda project was created using this template. The template also included the Widgets.Lambda.Tests project for testing. As done in the console application we need to install the following nuget packages

Amazon.Lambda.AspNetCoreSErver
Amazon.Lambda.Logging.AspNetCore (for cloudwatch logging)
AWSSDK.DynamoDBv2
AWSSDK.Extensions
AWSSDK.S3

The Widgets.Lambda project is essentially a .Net Core Web API project. AWS allows the whole project to be deployed into S3 and run in Lambda. This is configured using CloudFormation. More details can be found here:

https://aws.amazon.com/blogs/developer/running-serverless-asp-net-core-web-apis-with-amazon-lambda/

Using the AWS SDK in Visual Studio 2017, we created the Widgets.Lambda project using the AWS Lambda Web API template. Since our APIs will be connecting into DynamoDB, we needed to add the following to the startup.cs file. Again – look at the main github project for full code source.

public Startup(IConfiguration configuration)
{
    Configuration = configuration;
    AWSConfigs.AWSRegion = Configuration.GetSection("AWSConfiguration")["AWSRegion"];
    AWSConfigs.AWSProfileName = Configuration.GetSection("AWSConfiguration")["AWSProfileName"];
}

// This method gets called by the runtime. Use this method to add services to the container
public void ConfigureServices(IServiceCollection services)
{
    services.AddMvc();

    // Add DynamoDB
    services.AddAWSService();

    // Add S3 to the ASP.NET Core dependency injection framework.
    services.AddAWSService();
}

There are two api controllers in the Widgets.Lambda project. The S3ProxyController.cs is part of the AWS project template and shows how we can interact with the S3 service. The WidgetsController.cs is the one I’ve created which interacts with the DynamoDB service.

Working with DynamoDB

Though we created the Widgets.Data project that was used by the Widgets console, we do not use this in the Widgets.Lambda project. They are running two different .Net versions (Core 2 vs Classical 4.5) and at the time I was working on these separately. For the Widgets.Lambda project, the data access code can be found in the Data folder. There is a Repository.cs class that has the main CRUD functionalities.

The DynamoDB SDK for .NET provides a few different ways to interact with the database. These are:

Using Document Model
Using Object Persistence Model
Using Server Level API

https://docs.aws.amazon.com/mobile/sdkforxamarin/developerguide/dynamodb.html

I’ve tried to demonstrate some of these approaches in the Repository.cs class. In particular, I have multiple approaches to doing the reads (including query and scan). Refer to the github project and observe the Repository.cs class in the Data folder for each of these implementations. For more information on how these different approaches work, as well as some pro/con comparisons refer to my other post on Working with DynamoDB:

http://solidfish.com/aws-dynamodb-sdk-api-dot-net/

Setup of Web front-end using S3

The web front end is hosted on AWS S3. It is a single index.html static file with Javascript embedded to perform the AJAX calls into the AWS API Gateway as configured through CloudFormation. Those APIs are the Widgets.Lambda Web API methods implemented in the sections above. Below is a snippet of that index.html file:

...
  <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
  <script>
    $(document).ready(function () {
      ...
      $("#btnTest1").click(function () {
        ...
        $.getJSON(url, function (data) {
          var getByBatch = '<h2>Get top 3 items by batch using Object Persistence Model</h2>';
          if (data.getByBatch.length) {
            getByBatch += tableHeader;
            $.each(data.getByBatch, function (i, item) {
              getByBatch += `<tr><td>${item.id}</td><td>${item.name}</td><td>${item.description}</td><td>${item.created}</td><td>${item.price}</td><td>${JSON.stringify(item.types)}</td></tr>`;
            });
          }
          getByBatch += '</table>';
          $("#results").append(getByBatch);
        });
        ...
      });
      ...
      $("#btnSave").click(function () {
         ...
            var widget = {
              name: $("#widgetName").val(),
              price: $("#widgetPrice").val()
            };
            $.ajax({
              url: url,
              dataType: "json",
              contentType: "application/json;charset=utf-8",
              type: "POST",
              data: JSON.stringify(widget),
              success: function (result) {
                var html = `<h2>Successfully Created Widget</h2>`;
                html += `<table><tr><th>Number</th><th>Id</th></tr>`;
                html += `<tr><td>1</td><td>${result.original.id}</td></tr>`;
                html += `<tr><td>2</td><td>${result.lowLevel}</td></tr>`;
                html += `<tr><td>3</td><td>${result.documentModel}</td></tr>`;
                html += `<tr><td>4</td><td>${result.objectModel}</td></tr></table>`;
                $("#results").append(html);
              }
            });
          ...
        }
      });
    });
  </script>
<body>
  <h1>Widgets Test</h1>
  <p>
    This is a static front end page to test AWS Widgets Project
  </p>
  <button id="btnTest1">1.AllWidgets</button>
  <button id="btnTest2">2.CreateWidget</button>
  <button id="btnTest3">3.ReadWidget</button>
  <button id="btnTest4">4.UpdateWidget</button>
  <button id="btnTest5">5.DeleteWidget</button>
  <br />
  <p>
    <input type="text" id="widgetId" name="widgetId" placeholder="Enter Widget Id" hidden> &nbsp;
    <input type="text" id="widgetName" name="widgetName" placeholder="Enter Widget Name" hidden> &nbsp;
    <input type="number" id="widgetPrice" name="widgetPrice" placeholder="Enter Widget Price" hidden> &nbsp;
    <button id="btnSave" hidden>Submit</button>
  </p>
  <br />
  <div id="results"></div>
</body>

Publishing an Amazon CloudFormation

The Widgets.Lambda project is published into AWS using CloudFormation and the AWS SDK inside Visual Studio. CloudFormation will create a stack definition based on the serverless.template file found inside the Widgets.Lambda project. This template file contains the definitions for the Lambda and S3 services. The S3 is used for storing the actual Widgets.Lambda project binaries which will be executed by Lambda. Refer to the AWS CloudFormation document linked below for details on creating the template. A snippet of the file is shown below for setting up the resources.

"Resources" : {
    "ProxyFunction" : {
      "Type" : "AWS::Serverless::Function",
      "Properties": {
        "Handler": "Widgets.Lambda::Widgets.Lambda.LambdaEntryPoint::FunctionHandlerAsync",
        "Runtime": "dotnetcore2.0",
        "CodeUri": "",
        "MemorySize": 256,
        "Timeout": 30,
        "Role": null,
        "Policies": [ "AWSLambdaFullAccess" ],
        "Environment" : {
          "Variables" : {
            "AppS3Bucket" : { "Fn::If" : ["CreateS3Bucket", {"Ref":"Bucket"}, { "Ref" : "BucketName" } ] }
          }
        },
        "Events": {
          "PutResource": {
            "Type": "Api",
            "Properties": {
              "Path": "/{proxy+}",
              "Method": "ANY"
            }
          }
        }
      }
    },

Amazon Cognito

For user authentication AWS provides the Cognito service. This can use AWS managed accounts or OAuth support for third-party identity providers. More information about setting up Cognito can be found here:

https://github.com/awslabs/aws-serverless-workshops/tree/master/WebApplication/2_UserManagement